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Abstract — In this paper we propose a matched decoding 
scheme for convolutionally encoded transmission over intersym- 
bol interference (ISI) channels and devise a nonlinear trellis 
description. As an application we show that for coded continuous 
phase modulation (CPM) using a non-coherent receiver the 
number of states of the super trellis can be significantly reduced 
by means of a matched non-linear trellis encoder. 

Index Terms — ISI-channel; continuous phase modulation; con- 
volutionally encoded transmission; matched decoding; non- 
coherent differential detection; super trellis 

I. Introduction 

Convolutional encoding is an attractive encoding scheme 
due to its low latency compared to block encoding. When 
used for transmission with pulse amplitude modulation (PAM) 
schemes over an ISI-channel the receiver has to perform 
equalization and decoding either in two separated trellises 
or jointly in one super ttellis when due to strict latency 
consttaints interleaving and by this iterative equalization- 
decoding is prohibited. We propose a technique to merge the 
channel encoder with the ISI-channel. Instead of using ring 
convolutional codes to integrate the encoder into the M-ary 
ISI-channel as proposed in [1] we describe the ISI channel 
and the convolutional encoder by a single non-linear trellis 
encoder with binary delay elements. 

As an application continuous phase modulation (CPM) 
schemes are employed. It poses a class of power-efficient 
constant-envelope modulation schemes [2|. Due to the steady 
phase transitions the transmit signal of CPM has good spectral 
properties. Non-coherent receivers for CPM are relatively 
simple and straightforward to implement when compared to a 
coherent receiver. Using differential detection and a matched 
filter at the receiver one can easily retrieve the transmitted 
symbols by means of a decision-feedback equalization (DFE) 
or maximum-likelihood sequence estimation (MLSE) using 
the Viterbi algorithm (VA) comparable to a PAM transmis- 
sion over ISI-channels. The main disadvantage of differential 
detection is the spectral shaping of the channel noise. To 
counteract this performance loss, an additional discrete-time 
noise whitening filter is introduced which however further 
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extends the overall impulse response of the transmission 
system (5). 

We introduce coded transmission over ISI-channels using 
the general concept of serially concatenated convolutional 
encoder, mapper, modulator and ISI-channel in Section [TTJ 
At this point we show that the m-ary channel encoder and 
the A/-ary transmission scheme over the ISI-channel can be 
combined to a single m-ary trellis description representing 
a non-linear encoding. In order to facilitate description we 
resttict encoder and mapper to m = 2 and M — 2 n ; 
i.e. each output vector of a rate ^ convolutional encoder 
is mapped to one PAM-symbol. Section III shows that a 
complexity reduction can be achieved by combining multi- 
ple trellis states into hyperstates Q, (5}. We then describe 
non-coherent reception of CPM signals including differential 



detection, matched filtering and noise whitening in Section IV 



In Section [V] we verify via Monte-Carlo simulations that the 
proposed approach gives exactly the same performance with 
less states compared to the super trellis of the concatenation 
of convolutional encoder and the inherent continuous phase 
encoding of CPM (6), (7). 

II. Convolutionally Encoded Transmission over 

ISI-CHANNELS 

To introduce matched decoding (MD) we transform the 
convolutionally encoded transmission scheme step-by-step. 
For example consider the serial concatenation of a rate- 
(n" = l) binary convolutional encoder, a natural mapper, M- 
ary PAM transmission and an ISI-channel with L + 1 channel 
coefficients h[k] with k denoting the time index, cf. Fig. |T| 

A. Derivation of the Matched Encoder 

In the conventional approach one would process the receiver 
input signal first by a MLSE or a symbol-by-symbol trellis 
equalizer for the FIR filter h [k] and forward soft- or hard- 
output symbols of this ttellis equalization to the decoder for 
the channel code. But an optimum receiver however would 
perform MLSE over the super trellis decoding the binary 
channel encoder and the ISI channel impulse response h[k] 
of length L jointly. In a straight forward approach the super 
trellis would have Z enc ■ M L states, when Z enc is the number 
of states of the convolutional encoder. If the number of 
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Fig. 1: Concatenation of non-coherent CPM with a rate | 
convolutional encoder and an ISI-channel. 

output symbols from the encoder can be related to the size 
of the modulation alphabet M so that n = log 2 (A/) holds, 
the following complexity reduction can achieve exactly the 
same performance by a trellis description with less states. 
To see this, note that in each encoding step, n — K output 
symbols of the encoder are redundant and depend on K input 
symbols. E.g., in fig. [T]one of the two channel encoder output 
symbols contains no further information. We now show, how to 
combine the binary channel encoder with the Af-ary channel 
impulse response to form a single binary non-linear trellis 
encoder. 

First, we combine the P/S conversion and the mapper of 
Fig. [T] For clarity, we restrict ourselves to M = 4, but note 
that the concept easily extends to arbitrary M = 2™. In this 
example M — 4, i.e. n = 2, the upper branch corresponds 
to the most significant bit (MSB) whereas the lower branch 
describes the least significant bit (LSB). The natural mapping 
can be applied by multiplying the MSB by 2 and adding the 
LSB. The conversion from unipolar binary symbols c[k] into 
bipolar symbols b[k] within an alphabet of size M can be done 
with b[k] = (c[k] • 2) — 1. The resulting system is depicted in 
Fig. [2] Recall that the mod operation can be represented using 
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Fig. 2: Equivalent description of the convolutional encoding 
and ISI-channel (e.g. M = 4; natural mapping). 

the floor function. In terms of Gaussian notation we can thus 
write 



: mod n — x — 



(1) 



In addition we see that the main branch (after the sum- 
mation of MSB and LSB) has a multiplication and sum- 
mation which can be moved behind the convolution. With 
C = — J2k=o h[k](M — 1) and the Gauss representation of 
the modulo operation we can sketch the transmission system 
as depicted in Fig. [3] Note that now the convolution can be 
moved into the MSB branch and LSB branch, respectively, 



which enables to use binary delay elements instead of Af-ary 
ones. 




r[k] 



Fig. 3: Replacement of the mod 2 addition with the non-linear 
representation using floor function. 

This representation now has n independent binary branches 
which all depend on the same K input values (here n = 2 and 
K = 1). The calculations in each branch can be combined into 
a single non-linear filter. 

As an example we use the 4— state minimum free 
distance convolutional encoder with octal representation 
[#i,oct; ff2,oct] = [5 ct; 7 oct ] as binary generator polynomials 
for the MSB and LSB branch. The resulting, non-linear trellis 
encoder is depicted in Fig. [4] 
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Fig. 4: The matched encoder (ME) as a non-linear encoder 
representation of coded non-coherent CPM. 

B. Complexity Comparison 

The main advantage of matched encoding is the reduction 
of the convolution by the ISI-channel from an Af-ary input 
sequence into log 2 (Af) binary parallel convolutions in each 
branch. As the number of convolutions affect the calculation 
of metrics at the receiver but does not influence the number of 
resulting MLSE states we will now examine the complexity 
of trellis equalization for a matched decoding (MD) receiver 
and the traditional/serial super trellis decoding (STD). 

1) Super Trellis Decoding: In a super trellis we consider 
channel encoder states and channel states separately. The 
channel encoder is defined using generator polynomials with v 
binary memory elements resulting in Z enc — 2 V states for the 
trellis. The channel impulse response can be described using 
the impulse response h[k] with L + 1 filter coefficients. The 
ISI-channel with A/-ary input symbols has = M L states 
resulting in a total number of states in the super trellis of 

Zstd = Z enc ■ Z dYd = T ■ M L = r ■ 2(" L ) . (2) 



TABLE I: Number of states for non-coherent CPM transmis- 
sion with M = 4, n = 2 and for the super trellis representation 
and MD, respectively. 
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2) Matched Decoding: There are two differences compared 
to STD when considering the proposed matched encoding 
approach. First, the convolution with the channel impulse 
response is done with binary delay elements in contrast to M- 
ary elements. Second, as the MSB and LSB depend on each 
other (as of the channel encoder) not all state transitions are 
allowed anymore. As can be seen from Fig.|4]the total number 
of delay elements does not increase although we use binary 
delay elements, only. Thus, we still have 2 V possible states for 
the binary channel encoder (which is fully integrated into the 
non-linear encoder) and 2 L possible states for the convolution 
resulting in a total number of states of 



(3) 



Recall that for n — 2 there are two convolutions in parallel 
for the computation of the hypothesis. 

3) Comparison: The main advantage of MD compared to 
STD is the reduction of states without loss in performance. 
The resulting trellis still describes the super trellis but with 
less states. The gain of this state reduction can be calculated 
to 
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Table [I] summarizes several examples for different encoder size 
and channel lengths for the special case of n = 2 (M = 4). 
Obviously the gain increases with the length of the ISI- 
channel. 

III. Reduced-State Sequence Estimation 

We have shown that the super trellis of convolutionally en- 
coded transmission over ISI-channel can be represented using 
significantly less states by parallelizing the Af-ary convolution. 
At this point we can use reduced state sequence estimation 
(RSSE) to further reduce the number of states at the cost of 
small loss in Euclidean distance. In RSSE several MLSE states 
are combined into hyperstates p), ||5) which are then used 
for decoding. The partitioning is crucial to the performance 
of RSSE as it reduces the Euclidean distance. We w.o.l.o.g 



assume that the channel impulse response h[k] is minimum 
phase (by application of a proper all-pass filter) allowing use 
a partitioning comparable to that of delayed decision-feedback 
sequence estimation (DFSE) [8 |-pT|. 

In this work we apply this DFSE partitioning to use RSSE 
for MD of convolutionally encoded CPM with non-coherent 
differential detection and noise whitening as will be described 
below. 

IV. Non-Coherent CPM Reception 

Our application for MD is based on a CPM transmission 
over an AWGN channel. ISI results from the differential 
detection and a noise whitening filter which we will discuss 
shortly. The phase of a CPM transmit signal s(t) is given as 



(a,t) = 2irh ^ a[k] 

k— — oo 



g(t- kT)dt 



(5) 



2nh a[k]q{t-kT) 



k— — oo 

where h — | is the modulation index (p and q relative 
prime), g(t) is the frequency pulse of length L ■ T with 
the symbol duration T and a = {a[fc]} is a sequence of 
transmit symbols taken from the bipolar Af-ary alphabet 
{±1; ±3; . . . ; ±(Af - 1)} (M even). An integration of g(t) 
over time t gives the phase pulse q(t) with normalization 
q(t) = | Vt > LT. With an arbitrary phase-offset Oq and 
the signal energy per modulation interval E s , The equivalent 
complex baseband (ECB) transmit signal is given by 



s(t) = A /^exp{j (0(a,i) + 0o)} 



(6) 



explfinh ^ a[k]q(t - kT) + d 



A. Differential Detection 

Non-coherent demodulation for additive white Gaussian 
noise (AWGN) channels can be implemented using a differ- 
ential detection as described in p). In this paper a differ- 
ential detection for Af-ary partial-response (L > 1) CPM 
is used. The received signal r(t) is first band-limited, then 
the phase of the signal is extracted and unwrapped by a 
phase continuation (Phase unwrapping may be implemented 
by means of sampling at a sufficiently high frequency, reduc- 
tion of phase differences mod 2ir, and subsequent integration). 
Finally a differentiator provides the phase differences between 
two subsequent samples separated by X^. The receiver is 
shown in Fig. [5] (ECB-domain). For an ideal continuous- 
time differentiation Td — » holds. This demodulator fully 
inverts the non-linear CPM encoding. Hence a matched filter 
jg*(—t) for transition of continuous to T-spaced discrete time 
representation may be applied without loss of information on 
the sequence of data samples. The parameter 7 = —j= = is 

used for normalization with the energy of g(t) denoted as E g . 
Note, that for an AWGN channel the noise after the matched 



filter is non-white and non-Gaussian due to the differential 
demodulation. This issue is often referred to as FM-noise (or 
/ 2 -noise) (5). 
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Fig. 5: Non-coherent differential demodulation for CPM. 

B. Description of Intersymbol Interference 

As the modulation with differential demodulation only 
affects the noise power spectral density we can replace the 
CPM transmission over an AWGN channel with an equivalent 
PAM representation with non-white non-Gaussian noise as 
depicted in Fig. [6] An optimum receiver for non-coherent 

FM-noise/AWGN-noise 
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Fig. 6: PAM representation of CPM with differential demod- 
ulation (with substitute AWGN for derivation of theory). 

CPM would require a continuous-time whitening filter in front 
of the matched filter to decorrelate the FM-noise. As a noise 
whitening filter would not be stable we neglect the non-white 
and non-Gaussian characteristics of the noise at this point and 
instead develop a theory for AWGN-channel at first. In 1 3 1 
a noise-whitening filter is introduced after the "whitened" 
matched filter w.r.t AWGN channel which will be described 
in detail later on. 

In order to achieve a high bandwidth efficiency the common 
approach for CPM is to use a Raised Cosine pulse of length 
LT (L-RC pulse) with L > 1 for frequency impulse g(t) 
which results a) in ISI in the sequence d[k] of T-spaced 
samples of the Matched-Filter output (as G(f) does not satisfy 
the -^/Nyquist condition) and b) in less energy in the spectral 
side-slopes and thus higher spectral efficiency. Due to ISI 
equalization has to be performed at the receiver. The ISI can 
be described with the energy spectral density 3>gg(/) of the 
transmit pulse, given by 



<fgg(/)=7G(/)G*(/)= 7 |G(/)| 5 



and after T-spaced sampling, 

oo 



/ 



(7) 



(8) 



Obviously, 3> gg (/) does not fulfill the Nyquist criterion for 
L > 1. For L-RC CPM the ISI interacts with 2L-1 symbols 
due to $ gg [e j27r/T ]. 



At this point a Viterbi algorithm can be used to estimate 
the transmitted symbols but the noise power-spectral density 
is still non-white due to the FM-noise and the matched filter. 

C. "Whitened Matched Filter" for AWGN-Channel 

To optimize the performance we now investigate the 
whitened matched filter (WMF) [JT2J still assuming AWGN! 
This additional discrete-time whitening filter Hw(f) with 
an arbitrary phase <fi(f) and the power spectral density 
$ gg [e j27r/T ] is described by 



Hw(f) = 



(9) 



In combination with the receive filter G*(f) the whitening 
filter i?w(/) results in the whitened matched filter which is 
a y'Nyquist function. Using spectral decomposition of the Z- 
Transform $g g [z] = B[z] ■ S*[z* _1 ] we can then separate the 
power spectral density $ gg [<3 i27r f T ] into a causal minimum 
phase B[z] and a non-causal maximum phase part B*(z*^ 1 ). 
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(10) 



After sampling the overall discrete-time transmission can be 
summarized using a single filter jB[z] as depicted in Fig. [7] 
But due to the differential demodulation the noise is indeed 
not AWGN but FM-noise (/ 2 -noise). 

D. Noise Whitening 

In this paper we apply a discrete-time noise whitening filter 
F[z] after the "whitened" matched filter in order to reduce the 
/ 2 characteristics of the noise (see |3j and Fig. [7]). Exact noise 
whitening would correspond to integration which would cause 
stability problems. Therefor we use suboptimum filtering with 
finite length Lnw- A measurement of the noise characteristics 
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Fig. 7: Added T-spaced FIR noise whitening filter F[z\. 
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Fig. 8: Auto correlation <P[k] of the noise and impulse responds 
of the T-spaced noise whitening filter f[k] with Lnw = 10. 

shows that the autocorrelation <P[k\ depicted in Fig. [8] (left) 



clearly shows correlated noise. We minimize the error variance 
that results from a prediction filter with coefficients p[k]: 

r..... 2- 



We now investigate the results using MD in combination 



min E 

p[k] 



[k] — p[fc]0[/i — k] 



k=l 



(11) 



This minimization leads to the Yule- Walker equations with 
the noise prediction coefficients p[k]: 



^Tp[k]<p[K-k)=(f>[K\ Vfc= l,2,...,Ljv 



k=l 



The coefficients of the noise whitening filter are then 



F[z] -^o f[k] 



1 for k = 0; 

—p[k] for 1 < k < Lnw- 



(12) 



(13) 



In the right hand side picture of Fig. [8] the resulting filter 
coefficients for the correlated noise from Fig. [8] (left) are 
depicted. At the receiver we then have to equalize for ISI 
/i[fc]c^-»S[z]F[z]. Varying the length of the noise whitening 
filter Lnw, we can trade between the residual colorfulness of 
the noise and the complexity of the receiver (in the case of 
MLSE). 

V. Simulation Results 

First we will show that the results for MD are indeed 
exactly the same as for decoding in the super trellis, cf., 
Fig. [9] Here a 4-ary CPM transmission with modulation index 
h = 2 and L^pm = 3 is used. The lowpass at the receiver 
limits the signal to the B 99 9 % bandwidth of the CPM signal. 
The "whitened" matched filter for AWGN is of length 20, 
whereas the length of the noise whitening filter F[z] is a 
parameter. The overall ISI described by h[k) is therefore of 
length L = Lqpm + ^lnw — 1- As convolutional encoder, 
the [5 0Ct ; 7 oct ] code was used. It becomes clear that the 
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Fig. 9: Simulation results for coded non-coherent CPM with 
the encoder polynom [5 0Ct ; 7 0Ct ], M = 4, 3-RC pulse and 

h=i 

performance of STD is equal to that of MD. The simulations 
are conducted only for L N w = {0; 1} due to fact that STD 
gets overly complex for longer whitening filters very quickly, 
cf, Table ^ 



with RSSE. In Fig. [10] and Fig. [TT] the same coded CPM 
transmission with non-coherent reception scheme, Lnw = 
and Lnw = 2 but with 64-state rate-| encoder with 
[133 oct ; 171 oc t] is used. At the receiver decoding is done using 
a) serial decoding of channel impulse response and channel 
code b) MLSE decoding using full-state super trellis and c) 
matched decoding with RSSE. For a) we have two different 
approaches. The hard decision approach uses a decision- 
feedback sequence estimation (DFSE) with 4 or 16 states 
and a full-state Viterbi algorithm with 2 6 states to decode 
the convolutional encoder. The soft decision approach com- 
prises a symbol-by-symbol detection using the well-known 
BCJR algorithm (T3), and the soft-input Viterbi algorithm 
for channel decoding. One can see that even 4-state MD 
decoding supersedes the performance of serial decoding and 
RSSE converges STD with increasing number of states. 
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Fig. 10: Simulation results for coded non-coherent CPM with 
the encoder polynom [133 oct ; 171 oct ], M = 4, 3-RC pulse, 
h = j and Lnw = 0. 
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Fig. 11: Simulation results for coded non-coherent CPM with 
the encoder polynom [133 oct ; 171 oc t], M = 4, 3-RC pulse, 
h = \ and Lnw = 2. 

We now investigate the receiver complexity for concatenated 
equalization and channel decoding, MD and STD. We compare 
different channel encodings ISI channels defined by their num- 
ber of states. The target bit error rate is 10~ 3 and the receiver 



complexity is described by the number of states. For STD and 
MD with RSSE the receiver complexity is directly given by the 
number of states in the super trellis or can be defined, respec- 
tively. For concatenated equalization and decoding the receiver 
complexity is defined as the sum of states in the equalization 
and the decoding, i.e. C sem = Z cq + Z cnc ; C STD = Z cq ■ Z cnc 
with Z cq and Z cnc is the number of states in equalization trellis 
and the channel decoding trellis, represented. For RSSE the 
complexity depends on the partitioning so that Crsse = 2 r 
with arbitrary r. 



In Fig. 12 we compare MD for a transmission scheme with 
4 channel encoder states and 16 or 32 states in the ISI trellis 
(see also Fig. [9j. We can see again, that MD performs equally 
compared to STD, but with much less states. Additionally 
the receiver complexity for concatenated receiver structures 
are included. The best performance is achieve with DFSE 
equalizing a relatively long ISI of length 10. The channel 
encoding and the ISI-channel are described by their number 
of states and abbreviated with Z enc /Z c h a . 
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Fig. 12: Receiver complexity versus SNR for concatenated 
equalization and decoding, MD and STD. 

As our approach enables the use of RSSE we can compare 
the performance for arbitrary receiver complexity for the given 
target error rate. In Fig. 13 the results for MD and RSSE for a 
channel encoder with 16 or 64 states and an ISI trellis with 16 
or 64 states are compared. It becomes clear that for CPM with 
non-coherent reception the performance increases faster when 
using longer ISI-channel, i.e. longer noise whitening filters 
than with more states in the convolutional encoder. The figure 
shows clearly that, when using a 16 state channel encoder and 
an overall channel impulse response of length 10, only 4 states 
at the receiver are sufficient to supersede a channel encoding 
with more states and less ISI. 

VI. Conclusion 

In this paper we have shown that it is possible to reduce 
the number of states for the super trellis without loss of 
performance by transforming the M-ary channel convolution 
into log 2 (M) parallel binary convolutions. Here a coded non- 
coherent CPM transmission is used, but as several other 
non-interleaved transmission schemes {i.e. QAM over ISI- 
channel) can be represented as a separate channel encoder and 
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Fig. 13: Receiver complexity versus SNR for MD-RSSE. 

a channel impulse response this approach may be attractive 
for such schemes aswell. We showed that with MD the same 
performance can be achieved with much less effort. By using 
RSSE with DFSE-like partitioning the complexity can be 
reduced even further. 

The main drawback of the proposed MD approach is that 
it cannot be combined with interleaving between channel 
encoder and modulation, as in convolutional bit-interleaved 
coded modulation. 
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